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Abstract 


Educational design research (EDR) is described as a complex research approach. The 
challenges resulting from this complexity are typically described as procedural, whereas 
EDR might also be challenging for different reasons, specifically for early career 
researchers. Yet, challenging experiences may be noteworthy in the process of learning to 
do research and becoming a researcher. To explore this issue further, we engaged in a 
collaborative self-study, and conducted a narrative cross-case analysis of two PhD 
candidates’ experiences of engaging in EDR, focusing on challenges and learning 
outcomes. We find indications that the challenges of EDR might be related to EDR’s 
relatively new and minority position in educational sciences and the role a (early career) 
researcher needs to assume in EDR. Retrospectively, the challenges appear closely related 
to learning outcomes, which are described in terms of a more profound understanding of 
research (quality) and of oneself as a researcher. As such, insights gained by self-study of 
research practices provide a complementary perspective to existing literature on EDR and 
becoming a researcher. 
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1. Introduction 

Educational design research is described as a challenging research approach (e.g. Collins, Joseph, & 
Bielaczyc, 2004). Different aspects of this complexity are discussed in literature, typically tracing the origin 
of this complexity back to the multiple aims of educational design research, namely contributing to the 
general understanding of teaching and learning and creating a viable contextualized design to solve a local 
problem (Anderson & Shattuck, 2012). Some stress how educational design research might be especially 
challenging for early career researchers (Herrington, McKenney, Reeves, & Oliver, 2007), defined as 
researchers with up to ten years of experience (Andres, Bengtsen, Castano, Crossouard, Keefer, & Pyhalto, 
2015), including PhD students. The challenges of educational design research for early career researchers are 
described as procedural, whereas case studies (e.g. Akkerman, Bronkhorst, & Zitter, 2013) and experience 
suggest that educational design research might also be challenging for different reasons. While some studies 
suggest that such challenges can lead to PhD students experiencing dissonance (e.g. Wisker, Robinson, 
Trafford, Creighton, & Warns, 2003), others suggest that while challenges can be burdensome for PhD 
students, they can also be experienced as empowering (Stubb, Pyhalto, & Lonka, 2011), benefitting the 
learning process involved in becoming a researcher (Hall & Bums, 2009). 

Being early career researchers with experience in educational design research, we conducted a self- 
study exploring PhD candidates’ experiences with educational design research in terms of the challenges as 
well as the learning outcomes. Self-study is an unconventional and relatively unknown method, gaining 
popularity in research on teacher education (Zwart, Smit, & Admiraal, 2015) as a powerful way of providing 
insights complementary to those gained from other research methods (Bullough & Pinnegar, 2001; 
Loughran, 2007). As such, this article can be appreciated as a potentially thought-provoking example of 
using self-study methodology for studying researcher practices, illustrating the methodology’s potential and 
pitfalls to critically analyse early career researchers’ developing research practices. 


1.1 Educational design research (EDR) 

The origin of educational design research (EDR) is often traced back to the works of Brown (1992). 
Different authors use different terms, including design research, design-based research (or the abbreviation 
DBR) and design experiments; also, the specific methods used in EDR studies differ (Engestrom, 2011a; 
Reeves, Herrington, & Oliver, 2005). Barab and Squire (2004, p.2) typify EDR as “a series of approaches, 
with the intent of producing new theories, artifacts, and practices that account for and potentially impact 
learning and teaching in naturalistic settings”. In their review of the last decade of EDR research, Anderson 
and Shattuck (2012) characterize EDR as research situated in a real educational context, concentrating on 
testing a significant intervention, in collaboration with practitioner(s), informed by theories and an 
assessment of the local context as well as practices in other contexts. Accordingly, EDR uses mixed 
methods, involves multiple iterations to perfect the design, and requires a collaborative partnership between 
researcher(s) and practitioner(s), as it focuses on theory development and overcoming a problem in local 
practice, typically culminating in design principles. 

EDR is generally acclaimed as it is assumed to have the potential to enhance theoretical knowledge 
development while also having public educational value (van den Akker, 1999) and therefore resonates with 
wider calls for increasing the relevance and impact of educational research (Anderson & Shattuck, 2012). 
Baptista, Frick, Holley, Remmik, Tesch, and Akerlind (2015) describe how these calls for relevance are also 
being voiced in relation to research conducted as part of PhD dissertations, where the usefulness of the 
knowledge gained by the research is increasingly considered. Increasing attention to and application of EDR 
is demonstrated by the special issues devoted to this topic by leading educational research journals, including 
Journal of the Learning Sciences (Barab & Squire, 2004), Educational Researcher (Kelly, 2003) and 
Educational Psychologist (Sandoval & Bell, 2004). 
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More and more, EDR is not only applauded, but also critically assessed (Svihla, 2014). Several 
authors have pointed at potential weaknesses in EDR methodology (Dede, 2004; Shavelson, Phillips, Towne, 
& Feuer, 2003), questioning EDR’s potential to draw causal claims in natural settings and EDR’s tendency 
to generalize small scale studies (Kelly, 2004). Akkerman, Bronkhorst, and Zitter (2013) maintain that 
pursuing concurrent goals in EDR requires immediate and sometimes intuitive actions and decisions, as 
accepted systems of quality assurance are lacking. Barab and Squire (2004) highlight how EDR requires 
researchers to carefully balance insider and outsider perspectives: “If a researcher is intimately involved in 
the conceptualization, design, development, implementation, and re-searching of a pedagogical approach, 
then ensuring that researchers can make credible and trustworthy assertions is a challenge” (p. 10). 
Additionally, several scholars have addressed the challenges of analysis in EDR, given the large amounts of 
data that are usually collected (Collins, Joseph, & Bielaczyc, 2004; Kelly, 2004). 

Herrington, McKenney, Reeves, and Oliver (2007) have argued that specifically for beginning 
researchers, EDR might be too challenging. For one, the longitudinal nature of most design studies might 
extend the four-year period in which most PhD students are expected to complete their studies (Evans, 
2010). But even if the data collection itself can be completed, the richness of the data collected might extend 
the time and especially the skill needed for analysis. These challenges are presented and interpreted as 
procedural and thereby manageable, as can be deduced from Anderson and Shattuck’s (2012) solution in 
terms of multi-year multi-actor research agendas. Such solutions help advance EDR, but fail to appreciate 
the experience of engaging in EDR and its challenges for early career researchers. For instance, a case study 
of EDR conducted by a PhD candidate stresses the consequences in terms of feelings of insecurity and misfit 
of a seemingly procedural challenge—namely assuring quality in EDR studies (Akkerman et al., 2013). 
Recently, Castello, Kobayaski, McGinn, Pechar, Vekkaila, and Wisker (2015) have called attention to 
feelings of misfit as a PhD student, as they impact present and future professional aspirations and can lead 
students to abandoning the field of learning and instruction. Others have also cautioned against the 
consequences of feelings of dissonance as a PhD student (Wisker et al., 2003). 

1.2 Learning to do research 

In contrast, Lee and Roth (2003) argue that the learning potential of research actually lies in working 
through the challenges engaging in research is likely to generate. Others have also hinted at the learning 
potential of challenging circumstances for learning to do research, especially for early career researchers 
(e.g. Haigh, 2012; Hopwood, 2010). In such cases, learning is conceptualized as participation (Gonzalez- 
Ocampo et al., 2015) and learning outcomes are studied in terms of becoming a researcher, emphasizing not 
only the technical, but also the personal nature of learning to do research (Barnacle, 2005). “Transitioning 
[...] to the role of researcher is not as simple as acquiring a new set of skills and expanding one’s knowledge 
of scholarship” (Hall & Bums, 2009, p.53). Departing from this conceptualization, the biographical or 
narrative process of learning is usually studied, as well as the open-ended and potentially transformative 
outcomes for researcher identity of engaging in research (e.g. Lee & Roth, 2003). Such studies are in line 
with widespread calls for more in-depth, longitudinal research to explore what it means to become a 
researcher (Hall & Burns, 2009; Stubb et al., 2011). 

In terms of specific learning outcomes, Herrington and colleagues (2007) also report potential 
learning outcomes of conducting EDR specifically for PhD students. Most importantly, PhD students can 
leam to see practitioners as partners in research instead of beneficiaries of the outcomes of their research, 
thereby potentially increasing the impact of educational research on educational practice. Moreover, PhD 
students might benefit from learning about the ways in which EDR differs from other research methods. 
Also stressing the importance of awareness of the implications of methodological decisions made during 
research, Newbury (2002, p.156, emphasis added) states that “[pjerhaps most important is research students’ 
exposure to alternative approaches”. Similarly, Pallas (2001) emphasizes how preparing doctoral students 
for an essentially unpredictable future entails acquainting them with epistemological diversity as consumers 
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(i.e. reading research grounded in diverse epistemologies) as well as producers (i.e. engaging in research 
with diverse epistemologies). 

Summarizing, despite the fact that the challenges and learning potential of engaging in EDR for early 
career researchers are acknowledged, there is still a great deal to explore about the actual experience of 
engaging in EDR. In this study, we explore what challenges and learning outcomes PhD students experience 
when engaging in EDR , as such insight could not only support early career researchers and their supervisors 
in making their EDR studies successful, but also provide an in-depth insider perspective, informing a wider 
audience about what it means to become an (educational design) researcher. 


2, Methods 

2.1 Context of the study 

Cognizant of the differences in early career researcher education across countries (Andres, Bengtsen, 
Castano, Crossouard, Keefer, & Pyhalto, 2015), we detail the specific characteristics of PhD trajectories in 
educational and learning sciences in the Netherlands, where this study was conducted. First of all, PhD 
trajectories are jobs; PhD candidates do not pay tuition, but are paid for doing research. As such, the phrase 
‘PhD student’ is not used in Dutch, as candidates are not seen as students, although they are supervised by at 
least one full professor and one so-called ‘daily supervisor’ - usually an assistant or associate professor. 
Additionally, the PhD dissertation consists of four semi-independent articles, written in English and 
preferably published internationally. Hence, although coursework is included in this trajectory, it usually 
does not extend throughout the trajectory, as the focus of the four-year trajectory is the research project. 
Consequently, a completed (research) Master’s degree is a prerequisite to enter a four-year PhD trajectory. 


2.2 Self-study 

Given the exploratory and interpretative research aim, and conceptualizing PhD learning as 
participation, we chose to conduct a collaborative self-study, also referred to as auto-ethnography (e.g. Holt, 
2003). Self-study is typified as a methodology for studying professional practices that stems from a desire 
“to be more fully informed about the nature of a knowledge of practice” (Loughran, 2007, p.14). In the field 
of learning and instruction, self-study is an approach mainly used in research on teacher education and 
teaching (Zwart, Smit, & Admiraal, 2015). Self-study has become recognized as a powerful methodology to 
promote critical reflective attitudes, understand the relationship between theory and practice more 
profoundly, and develop knowledge from an insider perspective (Loughran, 2007; Petrarca & Bullock, 2014; 
Williams & Ritter, 2010). The increasing use of self-study, reflected in the creation of the self-study journal 
Studying Teacher Education, can be understood in the light of current debates on methodology—more 
specifically, debates on how to take into account the contextualization of human thinking and acting, and the 
importance of both outsider and insider understanding in research on teacher education (e.g., Hamilton, 
Smith, & Worthington, 2008) and in educational research in general (e.g., Maxwell, 2004a). 

2.3 Participants 

The authors of this paper are the two participants in this self-study. For clarity, we use pseudonyms 
and refer to them in the third person, and only use ‘we’ to refer to ourselves as authors of this article. At the 
time of the study, both participants were in the penultimate year of their PhD trajectory. They had started 
their PhD trajectories at the same university department in 2008, having completed the same research 
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Master’s degree in two consecutive cohorts. The full professors who supervised their PhD research differed, 
but the PhD candidates had the same daily supervisor. 

Mary was 25 years old at the time of data collection. In her EDR study, she originally aimed to 
design a tool to support the goal-relatedness of the Master’s thesis supervision by conducting group 
discussion meetings and individual interviews with five supervisors with a locally good reputation (see also 
de Kleijn, Meijer, Brekelmans, & Pilot, 2015). Erica was 28 years old. She studied how student teachers’ 
meaning-oriented learning and deliberate practice could be fostered by collaboratively re-designing two 
year-long courses in the teacher education program with two pairs of teacher educators, based on design 
principles developed in prior research 2 (see also Bronkhorst, Meijer, Koster, & Vermunt, 2011; Bronkhorst, 
Meijer, Koster, Akkerman, & Vermunt, 2013). Mary and Erica informally discussed their progress in their 
EDR studies and it seemed that they had quite different experiences, which they found striking given their 
similar background. This triggered the desire for a more systematic exploration of their experiences in EDR 
by means of a collaborative self-study. 

2.4 Interviews 

We assumed that a probing interview might help in the explication of challenges and learning 
outcomes, as self-narratives have been shown to be a powerful methodology for self-studies (Haigh, 2012). 
Such interviews require well established interview skills and can benefit from the interviewer and 
interviewee being acquainted (Lichtman, 2006). Therefore, the interviews were conducted by their daily 
supervisor, Christine, as she knew both Mary and Erica and had extensive experience in conducting 
qualitative research in general and open interviews specifically. 

Christine was asked to conduct individual in-depth interviews with the PhD candidates revolving 
around broadly defined topics: (1) the EDR research that they had conducted; (2) the challenges they had 
experienced in their research and how they had dealt with these; and (3) the learning outcomes in the process 
of becoming a researcher that they attributed to engaging in EDR. These themes were to be addressed 
longitudinally (i.e. in terms of their development over time). 

As a potential learning outcome of engaging in EDR concerns a different relationship with 
participants (Herrington et al., 2007), Mary and Erica had interviewed the participants in their EDR studies 
upon completing their studies. Selected fragments concerning the EDR participants’ perception of the PhD 
candidates from these interview were provided to Christine, to inform her about the participants’ 
perspectives. Christine designed an open interview structure (see Table 1) adhering to this input and these 
guiding principles. She explicitly used her knowledge of the PhD candidates to have them explicate more. 
Both interviews lasted about an hour and a half and were fully transcribed. 


2 Although departing from design principles, the approach to the collaborative design in Erica’s study can be 
characterized as a formative intervention (see also Bronkhorst et ah, 2013; Penuel, 2014). 
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Table 1. 

Interview themes and example questions 


Interview themes 

Example questions 

Engaging in EDR 

Can you explain your reasoning for designing the study the way you did? 

Would you (still) call it (design) research and why? 

Challenges 

experienced 

I can recall that this was challenging, at times. Can you tell me some more 

about that? 

How did you deal with this challenge? Was this a conscious choice? 

Becoming a 

researcher 

How would you describe yourself as a researcher? 

What did you learn by engaging in EDR? 

Additional probes 

used for all themes 

Would you do/have done it differently in the future/past? 

Can you give an example? 


2.5 Analysis 

A cross-case analysis of experiences of engaging in EDR was performed. This was preceded by a 
within-case, narrative, connective analysis (Maxwell, 2004b) of each PhD candidate individually. First, 
drawing on critical incident technique (Meijer, de Graaf, & Meirink, 2010), in both interviews fragments 
were selected where challenges and/or (learning) outcomes were substantially discussed. We verified our 
selections by scrutinizing the transcripts for words that indicated emotions (e.g. ‘doubt’), struggles (e.g. 
‘difficult’) and/or words that indicated changes (e.g. ‘different’) or time differences (e.g. ‘now’). Secondly, 
we traced each of these key experiences backwards and forwards: in the transcripts, we identified the 
processes by which they came about and how they subsequently developed, as well as factors or individuals 
that had influenced their origin, development or outcome. In order to triangulate the findings from these 
interviews with the PhD candidates, the interviews with the participants of both PhD candidates’ EDR 
studies were also scrutinized for confirming and discontinuing evidence. Thirdly, we compared and 
discussed our individual findings from the previous steps until a consensus on the relevant themes and their 
relationships was reached. The quality of this last step was enhanced by a ‘peer-debriefmg’ (Guba, 1981), in 
which a colleague, unfamiliar with the study, read the data and analysis and critically questioned the initial 
findings. 

Based on these steps, we created two descriptions which are presented chronologically in the results 
section in order to increase legibility and understanding for readers. These descriptions are based on and 
contain illustrative quotes from the interview transcripts of the interviews with the PhD candidates and of the 
data from their participants. We used these descriptions for the cross-case analysis. The cross-case analysis 
was sensitized by our theoretical framework, focusing on developing views on EDR methods and quality, the 
role of a researcher in EDR, and the process of becoming a researcher. 
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3. Results 

3.1 Mary 

Although an EDR study had been included in Mary’s research plan from the start, she kept 
postponing it. Her hesitation mainly resulted from the lack of guidelines, protocols or general conventions 
Mary found in the literature on EDR. She herself considered reliability, in terms of reproducibility, a key 
quality criterion for scientific research, for which clear and shared conventions were necessary. She 
generally preferred to be in control of what she was doing: 

“This is what makes me insecure and creates chaos in my head. Because there are no guidelines to 
hold on to and ‘everything goes And that I find very difficult. I obviously need boundaries and limitations. ” 

Additionally, at that time she thought research was about answering questions and proving or 
demonstrating theories. 

Yet, when an EDR study was no longer necessary for completing her thesis, she made the conscious 
decision to engage in EDR. She knew that EDR was well outside her comfort zone and thus would be fairly 
challenging for her, but she wanted to be(come) an all-round researcher and, being a PhD student, she 
counted on her supervisors’ support. 

For her EDR study, Mary invited expert thesis supervisors to three collaborative design meetings. 
She prepared these meetings extensively, but she reasoned that she could not completely control (“board 
up”) the research, as she sought her participants’ expertise on the topic of thesis supervision, for which she 
needed their ownership, expertise and creativity. This conscious lack of control over how the process 
unfolded was a recurring challenge for her, before and between these meetings. She also experienced an 
ethical dilemma in relation to the time investment she asked of her participants: 

“How will I ever tell them that they invested three times two hours, almost an entire working day? 
How will I ever tell them that I am not able to write [an article] about it? That it eventually does not lead to 

something that will be part of my dissertation. That was the biggest stress [factor], so to say. ” 

During the collaborative design meetings, she was immediately confronted with things that did not 
go as she had imagined, but she surprised herself by being able to adapt to unexpected circumstances 
successfully: 

“I thought: ‘I want to understand what is happening here. ’[...] At that time I did not worry at all, as 
in: ‘Gosh, where is this going? ’ It was more in looking back and when I started preparing the next meeting, 
that I panicked. Because that brought me back in the research mind set. But during the meetings I was 

mainly curious. ” 

The fact that she saw how she could relate her participants’ contributions to the literature greatly 
supported this process, as did their general enthusiasm: 

“They were also really captivated by the issue and also appreciated discussing it so much that my 
fear of ‘They are wasting their time here ’ lessened a little. And I also really observed during those meetings 
that [engaging in the study] brought about things for them as well. ” 

The planning, time-management and enthusiasm was also recognized by one of her participants, who 
indicated in the final interview: 

“Well, I really liked it, the way that you handled it. I mean, you are enthusiastic but also focused and 
flexible in the way you handled things. I found that very pleasant. ” (EDR participant) 

These meetings, the data collection, became a collaborative exploration and Mary’s role as a 
researcher changed in this process; she was seen more as an expert on the literature of supervision, which 
contrasted with her earlier experiences with questionnaires and interviews, where she felt like a “nitwit”. 
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Yet, it also meant that sometimes others took on the leadership role and determined the course of action in 
the meetings. Moreover, Mary did not only ask questions, but was also asked questions in return. She saw 
this as a sign of ownership on the part of her participants, necessary for the research, and welcomed it as 
such. 


Looking back, engaging in EDR brought about a number of changes for Mary. First, her view on 
research and research quality has shifted. She would no longer claim reproducibility is a key criterion for 
scientific research. Instead, transparency has become crucial and the dialogue with theory is now her first 
and foremost connotation with scientific research. Consequently, she would design future studies differently, 
leaving room to deviate from her original plan. Similarly, she would now say that research is about 
understanding and asking questions, rather than answering them. 

By engaging in EDR, she now has a clearer view on it, but she still perceives confusion between 
different perspectives on EDR: 

“I kind of have the impression that we have yet to agree on what [educational design research] 
actually is. Or at least that there are a lot of different perspectives on it. So it has become clearer for me 

what I would consider good design research. ” 

As her EDR was actually a quest for understanding and did not result in a design, she doubts whether 
she has actually engaged in EDR, an idea with which she now feels comfortable: 

“I don’t mind that at all. That was not my first priority, or goal. I consider such a design a means. 

What this resulted in—what I had not imagined in advance—is the [increased] understanding. ” 

Now, she would question the time investment asked of participants in completely controlled 
research: 


“Can you ask people to...what does it entail to ask people to participate in such a highly structured 
interview, in which there is hardly any room for their own input? In such a way that you as a researcher are 

only ‘taking’?” 

Moreover, she now sees the dialogue with theory, present already during data collection, as a key 
characteristic for research quality in general. 

These shifting perspectives also concern how she sees herself as a researcher. She no longer believes 
in the need to choose between the paradigms or ‘teams’ she perceived before, but feels comfortable as a 
multi-faceted and most of all curious researcher. She prefers certain types of research, including a preference 
for control and statistics, without considering these to be better types of research. 

“I immediately get the jibbers when I am assigned to a team, whether that is qualitative or 
quantitative. I like to think that the [research] question is leading, so to speak. The research design is then a 

means to answer the question, to put it like that. ” 

She now knows she is capable of doing different types of research to satisfy her curiosity. 


3.2 Erica 

Erica’s research goal was to study an intervention in educational practice, by having the educators 
(i.e. the participants in her study) experiment and explore the effects of that experimentation. The two main 
guiding principles for her research were that, first, educators should have agency in shaping educational 
design, as she believed that researcher control is not desirable nor possible in general, and, second, that the 
agency and expertise of the educators would actually make the design better. 

“So in that sense, / hoped that they would try out things for me, but I also hoped that the things they 
would then try out would be enriched with what they knew. So not just [based on] my theoretical 

knowledge. ” 
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Based on the literature she had read about EDR, she thought her research would be a somewhat 
structured research endeavour. She started her project combining ideas resulting from her Master’s with her 
intuitive ideas about how “the world works”. Initially, these ideas were almost contradictory. 

“Ifyou look at my research design, it really falls between two stools. The one is how I was educated, 
with large scale and more objective instruments on which I would have no further influence. And [on the 
other hand] apparently my intuitive ideas about how such a thing works and which data you need to collect 

for that. ” 

In her research design, she mainly attributed the valuable expertise required in this research to the 
educators, as she felt she had very little knowledge of the educational context, also calling herself a “nitwit”. 

Upon engaging in the research, Erica noticed how the process took its own turn, which she— 
contrary to her expectations—could not really characterize as structured nor cyclical, because every decision 
was grounded in a prior decision. Moreover, she noticed how she herself felt reluctant to exert agency. At 
some point, her participants asked her to share more of her knowledge and expertise, as they indicated in 
their interview: 

“At some point we said: [...] ‘We want to hear your opinion. [...] Especially [as it is] a different 
perspective. That is the added value. (EDR participant) 

This made her realize that her research would benefit from combining different expertise, including 
her own. She became more proactive and felt more at ease with steering the collaborative design according 
to her own agenda. She noticed how she recognized possibilities in which she could influence the educators’ 
engagement, relying on communicative techniques she normally did not associate with research (e.g. 
purposeful small talk) and which she had acquired elsewhere. 

“I had not considered [....] that I would apply those things. And that I would consider that they are 
part of research, [which] I had not really imagined. I thought it would be [...] much less interaction, or 

actually, much less personal. ” 

This led to ethical dilemmas as she questioned if it “was allowed” to act in this way deliberately in 
research. 


“But because I feel as though the values of education and research are different, and people always 
think that research is objective, and spotless and ethical and responsible, that makes it feel worse when you 
apply these [communicative techniques] as a researcher. ” 

All in all, this meant that her research design relied more and more on her intuition and less on what 
she assumed other educational researchers would prefer. Intuitively she thought her research “could not be 
done differently”, but rationally she feared the response of the educational and learning sciences community 
at large. Especially at conferences, Erica often realized how her research differed from research presented: 

“Actually at each research conference I attended, where a different type of research was presented, 
each time I thought: 7 do that completely differently ...why do I do that so differently? ’ Also because I had 

trouble indicating what it was exactly that / was doing, and why I thought it was important. ” 

This in turn led to doubts about herself as a researcher, which she hid from her supervisors, along 
with specific details about what she was really doing in her research. 

“During the year I had my doubts [about] if I was a researcher after all, or someone who 
participated. Or someone who was very meaningful for the educators, but who doesn’t amount to anything in 

the research context. ” 

Her participants recognized these doubts, as they indicated during their interview: 

“In that sense, I did not envy her [Erica], the past year. I was aware that she put herself in a difficidt 
position, by choosing this [research] approach. [...] Because it means that she has a lot to legitimize in the 

research domain. ” (EDR participant) 
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In dealing with these challenges, she relied on two strategies. First, she made an effort to explicate 
her intuition(s), which was rather implicit at the start of her study. Putting her rationale for the research in 
writing enabled her to have faith in the study’s value for research alongside its value for educational practice. 
Secondly, she purposefully shared this vision, first at conferences and later also with her supervisors. The 
feedback was positive, which made her trust her intuition more. 

“In that sense the AERA was also important. [...It was where] the keynote of Engestrom [2011b] 
took place and then I felt supported in: Okay, it might not be common what I assert, but there are people 

who ’d like to hear about it. ” 

As such, she feels that perhaps her vision on research has not changed much, but she now 
understands how more collaborative research designs, and the active role assigned to participants in these, 
benefit research next to educational practice, which was a crucial outcome for her. 

In terms of research and research quality, she assigns increased importance to (ecological) validity: 
measuring what one intends to measure in context. She describes how she might not feel comfortable doing 
educational research in decontextualized settings any more. In general, she would now argue that research 
control over a situation, often employed to yield reliability, might produce unnatural behaviour. She prefers 
to rely more on transparency and the available knowledge on human behaviour in designing her studies: 

“In that sense I started thinking less about how a research intervention ought to be and more about, 
‘What do we know about how people learn?”’ 

Looking back, Erica now thinks that her perspective on EDR or intervention research in general may 
not be shared by all, but is supported by some whom she holds high. Consequently, she would now call 
herself a researcher, albeit one with a specific perspective, namely: ‘We have as much to learn from practice 
as they do from us’. This in turn has increased her confidence towards the research community and her 
participants. 


3.3 Cross-case comparison 

3.3.1 Challenges of EDR 

Mary and Erica both describe how engaging in EDR was challenging, especially with respect to two 
archetypal characteristics of EDR: the cyclical process, including multiple iterations, and the relationship 
with participants. Both describe how their experience with EDR differs from how EDR is presented in the 
literature, specifically as being simplified and structured. Mary had expected this in advance, which was one 
of the main reasons for postponing her EDR study, whereas Erica was somewhat surprised, despite the fact 
that the complexity of EDR is described in the literature. Apparently, reading about EDR’s complexity— 
including the necessity of ongoing decisions and the challenges of balancing the multiple stakes and 
stakeholders involved—does not fully account nor prepare for the experience of EDR. Only when actually 
engaging in the research did it become clear to both PhD students what a cyclical research process entailed, 
indicating that learning by doing seems necessary for learning to conduct EDR. 

Despite having taken rather different approaches, Erica and Mary both experienced ethical dilemmas 
with respect to their role in relation to the role of the participants. The nature of their dilemmas differed. 
Erica struggled with the fact that she purposely invested in social interaction with her participants in order 
for them to be committed to her study, which she found ethically questionable in conducting research. The 
participants in her EDR study do not mention having noted such behaviour when describing the working 
relationship, which they felt was appropriate. Mary, on the other hand, struggled with the fact that her 
participants invested time in her study while she herself was not even sure about whether the study would 
result in a chapter in her dissertation. In contrast, the participants in her EDR study only mentioned how 
inspiring their participation in the EDR study had been for them. 
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3.3.2 Becoming a researcher 

EDR necessitated dealing with challenges and the resulting feeling of misfit. Mary had anticipated 
that she personally would not be a good fit with EDR; she questioned whether she would be able to cope 
with scant guidelines about how to carry out the study, mainly in light of her prior experience with more 
controlled research and personal preference for structured activities. Erica experienced a lack of fit between 
her EDR approach and what she considered to be the “general research community” as she questioned 
whether her collaborative approach to research, initially mainly informed by previous experiences outside 
academia, would be accepted and understood by the general educational research community, where other 
standards appeared to exist. 

Conducting their EDR studies involved working through these challenges and dealing with their 
insecurities, thereby fuelling reflections on becoming and being a researcher. Ironically in light of the 
insecurities, the most salient outcome is that in the end both PhD students describe a development from 
seeing themselves as ‘nitwits’ or novice researchers to being seen as an expert in their specific fields of 
study. In both cases, the participants in their EDR studies played an important part in the transition, as they 
explicitly asked the PhD candidates to take an expert role and to share their knowledge, even before the 
candidates themselves felt comfortable doing so. 

3.3.3 Quality of educational research 

Next to reflections on their own expertise, the challenges experienced in engaging in EDR triggered 
contemplations on the quality of educational research. Both PhD candidates described that after having 
engaged in EDR, they regarded transparency as one of the most important criteria for educational research, 
more than replicability. This appeared to result from the fact that replicability was impossible to achieve in 
EDR, which would imply that their EDR was not scientific. Yet, in hindsight, the PhD candidates do not 
question their research, as the ongoing dialogue with theory and the general quest for understanding made it 
scientific, in their opinion (and the acceptance of their papers in international journals supports that 
assertion). Both PhD candidates do, however, question the feasibility of replicability when it is 
operationalized in terms of complete researcher control. 


4, Conclusion and discussion 

This study departed from contrasting findings of previous studies concerning the challenges of PhD 
students when engaging in educational design research (EDR). For one, the challenges of EDR are described 
as procedural, but also as fundamental in the literature. Moreover, the consequences of dealing with such 
challenges are evaluated as burdensome, but also as empowering, benefitting the process of becoming a 
researcher. To explore these contrasting findings, this explorative study examined the experiences of PhD 
candidates engaging in EDR, focusing on challenges and learning outcomes. 

Our findings show that the challenges experienced concerned two typical aspects of EDR, namely 
the cyclical nature of an EDR process and the role of participants (e.g., Collins et al., 2004). More 
specifically, our analysis shows how the cyclical nature of EDR required ongoing decisions within limited 
time. Yet, it was not the procedural aspect of making these decisions, but the candidates’ awareness of the 
significant implications of these decisions for the research (cf. Newbury, 2002) that was experienced as 
challenging. Similarly, our findings reaffirm the potential EDR to support PhD students in learning to see 
practitioners as partners in research, rather than beneficiaries of the outcomes of their study (Herrington et 
ah, 2007). Yet, for the PhD candidates not interacting with participants, but balancing the multiple stakes and 
stakeholders involved - including the research community - was troubling and caused a lot of doubt and 
insecurity. It follows that while the iterative nature and the role of the participants are manageable for PhD 
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candidates engaging in EDR, the (perceived) conflicts with accepted quality standards of research can make 
them disruptive (as suggested by Akkerman, et al., 2013). 

The PhD candidates’ experiences in EDR contained both ambiguity and novelty, a combination that 
often proves to be quite challenging (cf. Wisker et al., 2003). However, the analysis illustrates how working 
through the challenges actually made the engagement in EDR educative, as this necessitated learning about 
alternative perspectives on and approaches to research and research quality. This resulted in a more 
elaborated perspective on EDR and research quality in general, and a more pronounced understanding of 
what kind of researcher the PhD candidates hoped to become. Finding that the learning outcomes are 
intertwined with the challenges extends previous findings on PhD learning, wherein PhD experiences were 
categorized as burdensome or empowering (Stubb et al., 2011), by illustrating that burdensome experiences 
can become empowering over time. More specifically, our findings show that in the end both PhD candidates 
describe having developed a more refined perspective on research quality, alongside a new understanding of 
what it means to be a researcher. The latter finding is in line with a conceptualization of PhD learning as a 
process of participation (Gonzalez-Ocampo et al., 2015) and learning outcomes in terms of becoming (Hall 
& Burns, 2009). This resonates with Pallas (2001) who, among others, stresses the importance of learning 
about and from epistemological diversity in research preparation. Hall and Burns (2009, p.61) draw attention 
to how learning about multiple and even contrasting epistemological approaches can inform PhD students 
about the researcher they want to become, as “becoming a professional researcher requires students to 
negotiate new identities and reconceptualize themselves both as people and professionals in addition to 
learning specific skills” (p.49). 

Generally speaking, the results of this study indicate that describing the challenges of EDR for early 
career researchers in terms of the necessary time and technical skills needed for the data collection and 
analysis, is insufficient. Alternatively, we would ascribe at least some of EDR’s complexity and learning 
potential for PhD students to two other aspects: EDR’s relatively new and minority position in educational 
sciences, and the role a researcher needs to assume to make EDR a success. 

First, EDR was not a mainstream research design in the Netherlands at the time of study and was 
only marginally discussed in doctorate curricula (cf. Wilhelm, Craig, Glover, Allen, & Huffman, 2000, for a 
similar discussion of qualitative research). The PhD candidates therefore did not have experience in 
designing, conducting or even reading about EDR. Moreover, some EDR characteristics differ from what the 
PhD candidates learned about research, as EDR aims to generate hypotheses rather than to prove them; is 
interactive rather than objective or distant; and relies on emergent instead of completely controlled designs 
(Collins et al., 2004). Secondly, the relationship with the participants in EDR also differs in many respects 
from other types of research, notably from those in which these PhD candidates were educated. The 
relationship with participants is a cornerstone of EDR and required Mary and Erica to take on a specific role 
as a researcher. Our findings indicate that two aspects of this role can be quite challenging. First, in EDR the 
participants and researchers are assumed to collaborate. It follows that part of the control over the research 
process is handed over to the participants (Edwards, Sebba, & Rickinson, 2007). Sharing the control over the 
research process is precisely what differentiates interventions that seek to build on participant agency and 
those that solely recognize researcher expertise, according to Engestrom (2011a). As the implications of 
shared research control have only recently become topic of debate, the PhD candidates had few examples or 
guidelines to build on. Secondly, the participants acknowledged and addressed the PhD researchers as 
knowledgeable or even experts on their research topic. This expert role not only disagreed with the PhD 
candidates’ perception of themselves as novices and their role as junior in the research domain, it it also 
implied that they as researchers might have had influence over the research content - something they have 
learned to avoid at all cost. These differences in perspectives highlight the merit of checking participant 
perspectives on the research, also as a way of quality assurance (see also Bronkhorst et al., 2013). 

Additionally, our analysis suggests that not only the causes, but also the experience of EDR’s 
complexity, especially for early career researchers, is not fully represented entirely in the literature. The EDR 
literature describes how EDR is and should be a cyclical process and a collaborative exploration for both 
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researcher and participants, but the specifics of such processes and collaborations are typically not discussed 
extensively. Akkerman and colleagues (2013) even suggest that there might be a difference between how 
EDR studies are reified as structured while “a lot of design researchers in praxis act differently” (p.422). For 
those learning to do research, being aware of potential differences between EDR practice and its reification 
might prove valuable. 


4.1 Limitations 

In light of calls for more in-depth research (Stubb et al., 2011), attending to experiences of engaging 
in EDR from a contextualized insider perspective was our main argumentation for choosing to conduct a 
self-study. While our findings highlight self-study’s possibilities for increased understanding of research 
practices, there are some pitfalls that deserve attention. For one, seeing as the content of the interviews was 
partly determined by the PhD candidates, ways in which EDR was not experienced as challenging or did not 
carry learning potential have received limited attention. In terms of interpretation of findings, distinguishing 
personally relevant findings from findings relevant for the field requires an outsider perspective in self-study. 
Therefore, we asked a colleague to be involved in verifying our analysis, supporting us in being less biased. 
Seeing as the anonymous reviewers also played a vital role in interpreting our findings from a wider 
perspective, we underscore the importance of debating self-study methods and findings publicly—in line 
with calls from the self-study community of teacher education (see, for instance, Loughran, 2007). 

4.2 Implications 

The findings illustrate how the experience of engaging in research in general, and EDR in particular, 
deserves more attention and support, which holds significant implications for designing and supervising 
early career researchers. In both cases studied, the EDR experience evolves from challenging to carrying 
learning potential, but not without effort. Therefore, for those debating whether or not EDR is (too) 
challenging for PhD students, we would like to stress how our study shows that engaging in EDR can 
challenge PhD students to develop and extend their methodological competence, to adapt and develop 
methodologies and perceive methodology as a field of study in itself. Few would not consider these to be 
valuable outcomes. Evans (2010), for instance, holds that they align with an extended understanding of 
professionality of early career researchers and Newbury (2002) claims that they lie at the core of 
methodological reflexivity, which he considers key to researcher preparation. However, for these challenges 
to be educative, our findings suggest there needs to be space, not only in terms of time and resources, but 
also conceptually and methodologically, and support for PhD students to work through such challenges. 
Unfortunately, such space is not always available in our age of efficiency and accountability (Biesta, 2010). 

In terms of support, our findings also draw attention to how engaging in EDR plays out differently in 
light of PhD students’ participation in other communities, echoing the results of Castello and colleagues 
(2015). For supervisors, it is important to realize that conducting EDR as an early career researcher brings 
about a range of insecurities and challenges that can differ depending on researcher personalities, previous 
experiences with research, and participation in other communities. Given the important role supervisors play 
in facilitating the successful completion of a PhD trajectory (Gonzalez-Ocampo et al., 2015), we suggest that 
supervisors need to adapt their supervision to the specific challenges that PhD students experience, referred 
to as “adaptivity” in the context of in the context of Master’s thesis supervision (de Kleijn, Bronkhorst, 
Meijer, Pilot, & Brekelmans, 2014). Yet, PhD students and supervisors should not avoid insecurities 
altogether, as they can carry learning potential. 

A final implication concerns the methodology used in our study. Typically, the professional practices 
under scrutiny in self-study concern teaching, with the exception of studies focusing on self-study as a 
method. There are very few examples in which research practices are the object of study (for an example, see 
Bronkhorst, van Rijswijk, Meijer, Koster, & Vermunt, 2013). In research on teacher education, self-study 
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has become recognized as a powerful methodology for teachers and teacher educators to promote local 
professional (knowledge) development, as well as develop relevant knowledge for the field from an insider 
perspective (Petrarca & Bullock, 2014; Williams & Ritter, 2010). Our experiences in this study suggest that 
invoking self-study to understand research practices can enrich our understanding in more than one way. Not 
only by means of our findings, foregrounding aspects of engaging in EDR undisclosed in the literature, but 
also by critically analysing and openly discussing the challenges involved in conducting research and 
becoming an educational researcher. 


Keypoints 

9 Studies PhD students’ experiences of engaging in educational design research. 

9 Finds specific EDR challenges and learning outcomes pertaining to early career researchers. 
9 Relates these challenges to EDR’s position in educational sciences. 

9 Advances self-study as a way for PhD students to learn about research practices. 
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